Cross-Lingual Classification of Crisis Data

Many citizens nowadays flock to social media during crises to share or acquire the latest information about the event. Due to the sheer volume of data typically circulated during such events, it is necessary to be able to efficiently filter out irrelevant posts, thus focusing attention on the posts that are truly relevant to the crisis. Current methods for classifying the relevance of posts to a crisis or set of crises typically struggle to deal with posts in different languages, and it is not viable during rapidly evolving crisis situations to train new models for each language. In this paper we test statistical and semantic classification approaches on cross-lingual datasets from 30 crisis events, consisting of posts written mainly in English, Spanish, and Italian. We experiment with scenarios where the model is trained on one language and tested on another, and where the data is translated to a single language. We show that the addition of semantic features extracted from external knowledge bases improve accuracy over a purely statistical model

Classifying Crises-Information Relevancy with Semantics

Social media platforms have become key portals for sharing and consuming information during crisis situations. However, humanitarian organisations and affected communities often struggle to sieve through the large volumes of data that are typically shared on such platforms during crises to determine which posts are truly relevant to the crisis, and which are not. Previous work on automatically classifying crisis information was mostly focused on using statistical features. However, such approaches tend to be inappropriate when processing data on a type of crisis that the model was not trained on, such as processing information about a train crash, whereas the classifier was trained on floods, earthquakes, and typhoons. In such cases, the model will need to be retrained, which is costly and time-consuming. In this paper, we explore the impact of semantics in classifying Twitter posts across same, and different, types of crises. We experiment with 26 crisis events, using a hybrid system that combines statistical features with various semantic features extracted from external knowledge bases. We show that adding semantic features has no noticeable benefit over statistical features when classifying same-type crises, whereas it enhances the classifier performance by up to 7.2% when classifying information about a new type of crisis

Incentivizing High Quality Crowdwork

We study the causal effects of financial incentives on the quality of crowdwork. We focus on performance-based payments (PBPs), bonus payments awarded to workers for producing high quality work. We design and run randomized behavioral experiments on the popular crowdsourcing platform Amazon Mechanical Turk with the goal of understanding when, where, and why PBPs help, identifying properties of the payment, payment structure, and the task itself that make them most effective. We provide examples of tasks for which PBPs do improve quality. For such tasks, the effectiveness of PBPs is not too sensitive to the threshold for quality required to receive the bonus, while the magnitude of the bonus must be large enough to make the reward salient. We also present examples of tasks for which PBPs do not improve quality. Our results suggest that for PBPs to improve quality, the task must be effort-responsive: the task must allow workers to produce higher quality work by exerting more effort. We also give a simple method to determine if a task is effort-responsive a priori. Furthermore, our experiments suggest that all payments on Mechanical Turk are, to some degree, implicitly performance-based in that workers believe their work may be rejected if their performance is sufficiently poor. Finally, we propose a new model of worker behavior that extends the standard principal-agent model from economics to include a worker's subjective beliefs about his likelihood of being paid, and show that the predictions of this model are in line with our experimental findings. This model may be useful as a foundation for theoretical studies of incentives in crowdsourcing markets.Comment: This is a preprint of an Article accepted for publication in WWW \c{opyright} 2015 International World Wide Web Conference Committe

Creating corroborated crisis reports from social media data through formal concept analysis

During a crisis citizens reach for their smart phones to report, comment and explore information surrounding the crisis. These actions often involve social media and this data forms a large repository of real-time, crisis related information. Law enforcement agencies and other first responders see this information as having untapped potential. That is, it has the capacity extend their situational awareness beyond the scope of a usual command and control centre. Despite this potential, the sheer volume, the speed at which it arrives, and unstructured nature of social media means that making sense of this data is not a trivial task and one that is not yet satisfactorily solved; both in crisis management and beyond. Therefore we propose a multi-stage process to extract meaning from this data that will provide relevant and near real-time information to command and control to assist in decision support. This process begins with the capture of real-time social media data, the development of specific LEA and crisis focused taxonomies for categorisation and entity extraction, the application of formal concept analysis for aggregation and corroboration and the presentation of this data via map-based and other visualisations. We demonstrate that this novel use of formal concept analysis in combination with context-based entity extraction has the potential to inform law enforcement and/or humanitarian responders about on-going crisis events using social media data in the context of the 2015 Nepal earthquake. Keywords : formal concept analysis, crisis management, disaster response, visualisation, entity extraction

Curiosity Killed the Cat, but Makes Crowdwork Better

Crowdsourcing systems are designed to elicit help from humans to accomplish tasks that are still difficult for computers. How to motivate workers to stay longer and/or perform better in crowdsourcing systems is a critical question for designers. Previous work have explored different motivational frameworks, both extrinsic and intrinsic. In this work, we examine the potential for curiosity as a new type of intrinsic motivational driver to incentivize crowd workers. We design crowdsourcing task interfaces that explicitly incorporate mechanisms to induce curiosity and conduct a set of experiments on Amazon’s Mechanical Turk. Our experiment results show that curiosity interventions improve worker retention without degrading performance, and the magnitude of the effects are influenced by both the personal characteristics of the worker and the nature of the task.Engineering and Applied Science

Intrinsic Elicitation : A Model and Design Approach for Games Collecting Human Subject Data

Applied games are increasingly used to collect human subject data such as people’s performance or attitudes. Games a ord a motive for data provision that poses a validity threat at the same time: as players enjoy winning the game, they are motivated to provide dishonest data if this holds a strategic in-game advantage. Current work on data collection game design doesn’t address this issue. We therefore propose a theoretical model of why people provide certain data in games, the Rational Game User Model. We derive a design approach for human subject data collection games that we call Intrinsic Elicitation: data collection should be integrated into the game’s mechanics such that honest responding is the necessary, strategically optimal, and least e ortful way to pursue the game’s goal. We illustrate the value of our approach with a sample analysis of the data collection game Urbanology

Crowdsourcing: A new tool for policy-making?

Crowdsourcing is rapidly evolving and applied in situations where ideas, labour, opinion or expertise of large groups of people are used. Crowdsourcing is now used in various policy-making initiatives; however, this use has usually focused on open collaboration platforms and specific stages of the policy process, such as agenda-setting and policy evaluations. Other forms of crowdsourcing have been neglected in policy-making, with a few exceptions. This article examines crowdsourcing as a tool for policy-making, and explores the nuances of the technology and its use and implications for different stages of the policy process. The article addresses questions surrounding the role of crowdsourcing and whether it can be considered as a policy tool or as a technological enabler and investigates the current trends and future directions of crowdsourcing. Keywords: Crowdsourcing, Public Policy, Policy Instrument, Policy Tool, Policy Process, Policy Cycle, Open Collaboration, Virtual Labour Markets, Tournaments, Competition

Situated crowdsourcing using a market model

Research is increasingly highlighting the potential for situated crowdsourcing to overcome some crucial limitations of online crowdsourcing. However, it remains unclear whether a situated crowdsourcing market can be sustained, and whether worker supply responds to price-setting in such a market. Our work is the first to systematically investigate workers ’ behaviour and response to economic incentives in a situated crowdsourcing market. We show that the market-based model is a sustainable approach to recruiting workers and obtaining situated crowdsourcing contributions. We also show that the price mechanism is a very effective tool for adjusting the supply of labour in a situated crowdsourcing market. Our work advances the body of work investigating situated crowdsourcing. Author Keywords Crowdsourcing; virtual currency; market; situated technologies. ACM Classification Keywords H.5.m. Information interfaces and presentation (e.g., HCI)